Method of processing an infrared image, infrared image capturing system and computer readable medium

ABSTRACT

A system and method for processing an infrared image. The infrared image is processed to provide a background portion of the infrared image and a detail portion of the infrared image. The background portion and/or the detail portion is scaled to provide a level of the detail portion relative to a level of the background portion. The background portion and the detail portion are merged after the scaling to provide a processed infrared image. The processed infrared image is stored.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is the national phase under 35 U.S.C. §371 ofPCT/SE2007/000669 filed 9 Jul. 2007.

TECHNICAL FIELD

The present invention relates generally to infrared imaging systems and,more particularly, to image processing techniques for infrared imagingsystems and methods for controlling the display of an infrared image.

BACKGROUND

Infrared cameras are utilized in a variety of imaging applications tocapture infrared images. These captured infrared images may be grainy(e.g., noisy) or lack sufficient detail and, therefore to some extent,processing techniques may be applied to suppress unwanted features, suchas noise, and/or refine the captured infrared images.

In general, infrared cameras may have to deal with two oftencontradictory signal characteristics. On the one hand, a digitizedinfrared image may have a large dynamic range, which may be measured indigital units (e.g., on the order of tens of thousands of digitalunits). Within this dynamic range, some faint detail might be of greatimportance to the user and which, in order to be visible to the user ona display, may require for example the application of a contrastenhancement filter. On the other hand, an infrared image may also sufferfrom poor signal to noise ratio (SNR) and consequently, contrastenhancement might make the image less useful because an enhancement tothe contrast may result also in the amplification of the noise.Typically, some kind of noise filter may be applied to the infraredsignal, but finding the right settings for the noise filter for aparticular imager, for a particular scene, and for a particularapplication may be time consuming. Therefore, minimizing the user's timefor finding the optimal setting, under which some detail or target ofchoice may best be viewed, may be very beneficial for a number ofdifferent types of applications.

A drawback of a conventional infrared camera is that a user is notallowed to control these processing techniques during capture of theimage or the optimal settings may be difficult to determine by the user.Consequently, from a user's perspective, the result is a less thandesirable image being captured and displayed. Furthermore, automaticallysetting various parameters may be difficult. For example, for two imageswith similar signal properties (e.g., SNR, dynamic range, etc.), thefainter details may be essential for classifying a target or may just beclutter that the user would rather suppress than enhance.

As a result, there is a need for improved techniques for providing imageprocessing techniques and/or user-controllable settings for infraredcameras.

SUMMARY

Systems and methods disclosed herein, in accordance with one or moreembodiments of the present invention, provide image processingtechniques for images captured by infrared sensors (e.g., infraredcameras) that may improve image quality. For example, in accordance withan embodiment of the present invention, image processing algorithms aredisclosed to separate an image signal into at least two parts, which maybe separately scaled before merging of the two parts to produce anoutput image. Furthermore, in accordance with an embodiment of thepresent invention, two matched tables of values may be generated for theimage processing algorithms for continuous transitioning from an imagewith a low level of detail to an image with a high level of detail andvice versa. Consequently, for example, a user may operate a controlfeature (e.g., a control device) to adjust the image processingalgorithms to control and set details of an image to a desirable level.

More specifically, in accordance with an embodiment of the presentinvention, a method of processing an infrared image includes processingthe infrared image to provide a background portion of the infrared imageand a detail portion of the infrared image; scaling the backgroundportion and/or the detail portion to provide a level of the detailportion relative to a level of the background portion; merging thebackground portion and the detail portion after the scaling to provide aprocessed infrared image; and storing the processed infrared image.

In accordance with another embodiment of the present invention, aninfrared image capturing system includes an image capture componentadapted to capture an infrared image; a processing component; and amemory component adapted to store information to control the processingcomponent to perform infrared image processing. The infrared imageprocessing includes filtering the infrared image to provide a backgroundportion of the infrared image and a detail portion of the infraredimage; scaling the background portion and/or the detail portion toprovide a level of the detail portion relative to a level of thebackground portion; merging the background portion and the detailportion after the scaling to provide a processed infrared image; andstoring the processed infrared image.

In accordance with another embodiment of the present invention, acomputer-readable medium on which is stored information for performing amethod, the method includes filtering an infrared image to provide abackground portion of the infrared image and a detail portion of theinfrared image; scaling the background portion and/or the detail portionto provide a level of the detail portion relative to a level of thebackground portion; performing histogram equalization to the backgroundportion; merging the background portion and the detail portion after thescaling to provide a processed infrared image; and storing the processedinfrared image.

The scope of the invention is defined by the claims, which areincorporated into this section by reference. A more completeunderstanding of embodiments of the present invention will be affordedto those skilled in the art, as well as a realization of additionaladvantages thereof, by a consideration of the following detaileddescription of one or more embodiments. Reference will be made to theappended sheets of drawings that will first be described briefly.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of an image capturing system in accordancewith an embodiment of the present invention.

FIG. 2 shows a method for improving image quality in accordance with anembodiment of the present invention.

FIG. 3 shows a block diagram of a method for processing an image inaccordance with an embodiment of the present invention.

FIGS. 4 a and 4 b show graphical representations of parameter settingsin accordance with an embodiment of the present invention.

FIG. 5 shows a front view of a hand control unit in accordance with anembodiment of the present invention.

FIG. 6 shows a block diagram of a method for processing an image inaccordance with an embodiment of the present invention.

FIG. 7 shows a graphical representation of a variable for processing animage, such as with the method of FIG. 6, in accordance with anembodiment of the present invention.

Embodiments of the present invention and their advantages are bestunderstood by referring to the detailed description that follows. Itshould be appreciated that like reference numerals are used to identifylike elements illustrated in one or more of the figures.

DETAILED DESCRIPTION

Systems and methods are disclosed herein, in accordance with one or moreembodiments of the present invention, to provide image processingalgorithms for images captured by infrared imaging systems, which mayimprove image detail and quality by allowing a user to selectively scale(e.g., enhance and/or diminish) one or more parts of the capturedinfrared image to a desired level. For example, in accordance with anembodiment of the present invention, an image signal provided by aninfrared camera may be separated into a background image (e.g., a lowspatial frequency, high amplitude portion of the image signal) and adetail image (e.g., a high spatial frequency, low amplitude portion ofthe image signal). The background image and/or the detail image, forexample, may then be scaled separately before they are merged to producean output image or stored for later display. As disclosed furtherherein, one or more embodiments of the present invention may providecertain advantages over conventional infrared imaging systems (e.g.,infrared cameras), such as for example, by providing one or more usercontrols that may reduce the complexity of choosing infrared camerasettings for the image processing algorithms (e.g., an optimal set ofparameters for a particular type of scene, such as a low light scenehaving high frequency noise that decreases overall image quality).

In accordance with an embodiment of the present invention, FIG. 1 showsa block diagram illustrating an image capturing system 100 for capturingand processing infrared images. Image capturing system 100 comprises, inone embodiment, a processing component 110, a memory component 120, animage capture component 130, a control component 140, a displaycomponent 150, and optionally a sensing component 160.

Image capturing system 100 may represent an infrared imaging device,such as an infrared camera, to capture images, such as an image 170.Image capturing system 100 may represent any type of infrared camera,which for example detects infrared radiation and provides representativedata (e.g., one or more snapshots or video infrared images). Forexample, image capturing system 100 may represent an infrared camerathat is directed to the near, middle, and/or far infrared spectrums.Image capturing system 100 may comprise a portable device and may beincorporated, for example, into a vehicle (e.g., an automobile or othertype of land-based vehicle, an aircraft, or a spacecraft) or anon-mobile installation requiring infrared images to be stored and/ordisplayed.

Processing component 110 comprises, in one embodiment, a microprocessor,a single-core processor, a multi-core processor, a microcontroller, alogic device (e.g., a programmable logic device configured to performprocessing functions), a digital signal processing (DSP) device etc.Processing component 110 is adapted to interface and communicate withcomponents 120, 130, 140, and 150 to thereby perform method andprocessing steps in a manner as described herein. Processing component110 may further comprise a Adaptive Contrast (AC) filtering module 112that is adapted to implement a AC algorithm (e.g., a AC filteringalgorithm), which will be described in greater detail herein. Processingcomponent 110 may also be adapted to perform various other types ofimage processing algorithms including scaling the signal of one or moreparts of the image, either as part of or separate from the AC algorithm,in a manner as will be described herein.

It should be appreciated that AC filtering module 112 may be integratedin software and/or hardware as part of processing component 110, or code(e.g., software or configuration data) for the AC filtering module 112may be stored in memory component 120. Moreover, for example, inaccordance with an embodiment of the present invention, embodiments ofthe AC algorithm disclosed herein may be stored by a separatecomputer-readable medium (e.g., a memory such as a hard drive, a compactdisk, a digital video disk, or a flash memory) to be executed by acomputer (e.g., a logic or processor-based system) to perform variousmethods disclosed herein. As an example, the computer-readable mediummay be portable and/or located separate from image capturing system 100,with the stored AC algorithm provided to image capturing system 100 bycoupling the computer-readable medium to image capturing system 100and/or by image capturing system 100 downloading (e.g., via a wired orwireless link) the AC algorithm from the computer-readable medium.

Memory component 120 comprises, in one embodiment, one or more memorydevices to store data and information. The memory device may compriseone or more various types of memory devices including volatile andnon-volatile memory devices, such as RAM (Random Access Memory), ROM(Read-Only Memory), EEPROM (Electrically-Erasable Read-Only Memory),flash memory, etc. Processing component 110 is adapted to executesoftware stored in memory component 120 so as to perform method andprocess operations described herein.

Image capture component 130 comprises, in one embodiment, one or moreinfrared sensors (e.g., any type of infrared detector, such as a focalplane array) for capturing infrared image signals representative of animage, such as image 170. In one example, the infrared sensors of imagecapture component 130 provide for representing (e.g., converting) thecaptured image signal as digital data (e.g., via an analog-to-digitalconverter included as part of the infrared sensor or separate from theinfrared sensor as part of image capturing system 100). Processingcomponent 110 may be adapted to process the infrared image signals(e.g., to provide processed image data), store the infrared imagesignals or image data in memory component 120, and/or retrieve storedinfrared image signals from memory component 120. For example,processing component 110 may be adapted to process infrared imagesignals stored in memory component 120 to provide image data (e.g.,captured and/or processed infrared image data).

Control component 140 comprises, in one embodiment, a user input device,such as a rotatable knob (e.g., potentiometer), push buttons, slide bar,keyboard, etc., that is adapted to generate an input control signal.Processing component 110 may be adapted to sense control input signalsfrom control component 140 and respond to any sensed control inputsignals received therefrom. Processing component 110 may be adapted tointerpret the control input signal as a value, which will be discussedin greater detail herein.

In one embodiment, control component 140 may comprise a generally knowncontrol unit 500 (e.g., a wired or wireless handheld control unit)having push buttons adapted to interface with a user and receive userinput control values, as shown in FIG. 5. For example, the push buttonsof control unit 500 may be used to control various functions of imagecapturing system 100, such as autofocus, menu enable and selection,field of view, brightness, contrast, and/or various other features aswould be understood by one skilled in the art. Furthermore in accordancewith an embodiment, one or more of these push buttons may be used toinput values (e.g., a AC value) for the AC algorithm. For example, pushbuttons 502(1) and 502(2), which may be used to adjust contrast in onemode of image capturing system 100, may be also used to adjust the ACvalue (e.g., increase or decrease the setting for the AC value) inanother mode of image capturing system 100 as discussed further herein.

Display component 150 comprises, in one embodiment, an image displaydevice (e.g., a liquid crystal display (LCD)) or various other types ofgenerally known video displays or monitors. Processing component 110 maybe adapted to display image data and information on display component150. Processing component 110 may also be adapted to retrieve image dataand information from memory component 120 and display any retrievedimage data and information on display component 150. Display component150 may comprise display electronics, which may be utilized byprocessing component 110 to display image data and information (e.g.,infrared images). Display component 150 may receive image data andinformation directly from image capture component 130 via processingcomponent 110, or the image data and information may be transferred frommemory component 120 via processing component 110.

Optional sensing component 160 comprises, in one embodiment, one or morevarious types of sensors, depending upon the desired application orimplementation requirements, which provide information to processingcomponent 110. Processing component 110 may be adapted to communicatewith sensing component 160 (e.g., by receiving sensor information fromsensing component 160) and with image capture component 130 (e.g., byreceiving data from image capture component 130 and providing and/orreceiving command, control or other information to and/or from othercomponents of image capturing system 100).

In various embodiments, sensing component 160 may provide informationregarding environmental conditions, such as outside temperature,lighting conditions (e.g., day, night, dusk, and/or dawn), humiditylevel, specific weather conditions (e.g., sun, rain, and/or snow),distance (e.g., laser rangefinder), and/or whether a tunnel, a coveredparking garage, or other type of enclosure has been entered or exited.Sensing component 160 may represent conventional sensors as would beknown by one skilled in the art for monitoring various conditions (e.g.,environmental conditions) that may have an affect (e.g., on the imageappearance) on the data provided by image capture component 130.

In some embodiments, optional sensing component 160 may comprise devicesthat relay information to processing component 110 via wirelesscommunication. For example, sensing component 160 may be adapted toreceive information from a satellite, through a local broadcast (e.g.,radio frequency) transmission, through a mobile or cellular networkand/or through information beacons in an infrastructure (e.g., atransportation or highway information beacon infrastructure) or variousother wired or wireless techniques.

In various embodiments, components of image capturing system 100 may becombined and/or implemented or not, as desired or depending upon theapplication or requirements, with image capturing system 100representing various functional blocks of a system. For example,processing component 110 may be combined with memory component 120,image capture component 130, display component 150 and/or sensingcomponent 160. In another example, processing component 110 may becombined with image capture component 130 with only certain functions ofprocessing component 110 performed by circuitry (e.g., a processor, amicroprocessor, a logic device, a microcontroller, etc.) within imagecapture component 130.

In accordance with an embodiment of the present invention, FIG. 2 showsa method 200 that may improve image quality and/or detail by allowing auser to selectively process (e.g., enhance and/or diminish) one or moreparts of the captured infrared image to a desired level. For purposes ofsimplifying the following discussion of FIG. 2, reference will be madeto image capturing system 100 of FIG. 1 as an example of a system thatmay perform method 200.

Referring to FIG. 2, an image (e.g., infrared image signal) is captured(block 210). In one embodiment, upon user input command, processingcomponent 110 induces (e.g., causes) image capture component 130 tocapture image 170. Processing component 110 receives the captured imagefrom image capture component 130 and optionally stores (block 212) thecaptured image in memory component 120 for processing.

The image is processed (block 220), wherein the processing may compriseadjusting (e.g., improving) image quality and detail. In one embodiment,processing component 110 either directly processes the image captured(block 210) or optionally retrieves the captured image stored in memorycomponent 120 (block 212) and then processes the image for improveddetail and quality (e.g., based on user input via control component 140)in a manner as will be discussed in greater detail herein.

The processed image is stored (block 222) and may be displayed (block230). In one embodiment, processing component 110 stores the processedimage in memory component 120 for displaying and/or processing component110 retrieves the processed image stored in memory component 120 anddisplays the processed image on display component 150 for viewing by auser.

A determination may be optionally made (block 240) as to whether acontrol input from a user is sensed. In one embodiment, if processingcomponent 110 senses a control input from a user via control component140, then processing component 110 obtains the sensed input value fromthe control component 140, and the method returns to block 220 so thatprocessing component 110 can process (e.g., re-process) the image inreference to the obtained sensed input control value (e.g., AC value).Otherwise, if processing component 110 does not sense a control inputfrom the user via control component 140, then the method returns to step230 so that processing component 110 can continue to display thepreviously processed image. These features will be discussed in greaterdetail herein.

In accordance with an embodiment of the present invention, an imageprocessing algorithm is adapted to separate an infrared image signal(e.g., captured infrared image) into at least two parts. This imageprocessing algorithm is referred to herein as a Adaptive Contrast (AC)algorithm (or AC filtering algorithm), which for example may be utilizedas a filter for filtering portions (e.g., detail image and/or backgroundimage) of an infrared image to provide at least a first part and asecond part.

In one embodiment, a first part of the image signal includes abackground image part comprising a low spatial frequency high amplitudeportion of an image. In one simplified example, a low pass filter (e.g.,low pass filter algorithm) may be utilized to isolate the low spatialfrequency high amplitude portion of the image signal (e.g., infraredimage signal).

In one embodiment, a second part of the image signal includes a detailedimage part comprising a high spatial frequency low amplitude portion ofan image. In one simplified example, a high pass filter (e.g., high passfilter algorithm) may be utilized to isolate the high spatial frequencylow amplitude portion of the image signal (e.g., infrared image signal).Alternatively, the second part may be derived from the image signal andthe first part of the image signal, such as by subtracting the firstpart from the image signal.

In one embodiment, the two parts (e.g., first and second parts) of theimage signal may be separately scaled with, for example, the ACalgorithm before merging to produce an output image. For example, thefirst part may be scaled or the second part may be scaled or both thefirst and second parts may be scaled. This allows the system to outputan image where fine details are visible and tunable even in a highdynamic range scene. The AC filter algorithm performance may becontrolled by a user via a number of parameters, which will be discussedin greater detail herein, to determine for example an optimal set ofparameters. In some instances as an example, if an image appears lessuseful or degraded by some degree due to noise, then one of the parts ofthe image, such as the detailed part, may be suppressed rather thanamplified to suppress the noise in the merged image to improve imagequality.

In accordance with an embodiment of the present invention, the ACalgorithm may be controlled by the following five variables.

Variable 1 (N) represents a kernel size of a non-linear low pass filteradapted to separate the detailed image part and the background imagepart. In the case for non-square kernels, a variable of M×N may be usedinstead, where M and N are different values.

Variable 2 (σ_(R)) represents a standard deviation of the radiometricGaussian, which defines an amplitude of the detailed image portion (orpart).

Variable 3 (σ_(S)) represents a standard deviation of the spatialGaussian, which defines a spatial reach of the non-linear low passfilter within the kernel size N (or M×N).

Variable 4 (D_(Range)) represents a dynamic range of the detailed imagepart of the signal in the output image (e.g., merged image).

Variable 5 (B_(Range)) represents a dynamic range of the backgroundimage part of the signal in the output image (e.g., merged image).

In some systems in accordance with an embodiment, variables N and σ_(S)may be held constant. Accordingly, in a fixed available output range(e.g., 8-bits or 256 colors of a grey-scale), D_(Range) may comprise afunction of B_(Range) as set forth below in equation (1). Thus, in thisembodiment, two parameters may be controlled: σ_(R) and the ratio ofD_(Range) over B_(Range), where the ratio may be referred to as thedetailed image part to the background image part (e.g., Detail toBackground Ratio or D2BR) as set forth below in equation (2).D_(Range)=256−B_(Range)  (1)D2BR=D_(Range)/B_(Range)  (2)

It should be appreciated that the manner in which these parameters areset will generally change the appearance of the image, and varioussettings of these parameters may be applied for different scene types.

In accordance with an embodiment of the present invention, the variableσ_(R) comprises a threshold. For example, when a low pass filter (LPF)is applied to a pixel of an image, the filtered value comprises aweighted mean of its neighboring pixels within a spatial range (e.g., 9by 9 pixels). In one example, a particular low pass filter that may beused to define the variable σ_(R) as a threshold comprises the generallyknown Bilateral Filter. Neighboring pixels having a value thatsignificantly differs, from the pixel being filtered by more digitalunits than the variable σ_(R) threshold, may not be used in theaveraging, such as by being given a weight of zero or close to zero forthe averaging (e.g., the result of which may be to inhibit the low passfilter from blurring across edges of the captured image).

In accordance with an embodiment of the present invention, D2BR (Detailto Background Ratio) comprises a ratio as set forth herein in equation(2). Depending on the type of low pass filter used and settings thereof,the dynamic range of the detailed part of the image may be measured(e.g., defined). In one example, the actual dynamic range may bemeasured, or when using the Bilateral filter, the maximum range may beproportional to the variable σ_(R). Given a dynamic range of the details(e.g., the second part or detailed image portion), referred to herein asDetail Dynamic Range (DDR), the gain that needs to be applied to thebackground from the D2BR setting may be calculated.

For example, given a dynamic range of the background (e.g., the firstpart or background portion), referred to herein as Background DynamicRange (BDR), a Background Gain (BG) may be set to obtain a desired ratiobetween details and background as set forth below in equation (3).Alternatively, an inverse of BG, such as 1/BG, may be applied in asimilar fashion to the details (e.g., the second part or detailed imageportion) to achieve a similar result and provide the desired ratiobetween details and background. Therefore, in one embodiment, thevariable σ_(R) may be adapted to control the background imagecomposition and the detailed image composition, while D2BR may be usedto calculate the gain that needs to be applied to the background (or tothe detail portion) to obtain the desired ratio of details andbackground. To maintain smooth transition from image smoothing to detailenhancement the tables are created in such a way that less and lesssmoothing is applied until a set of parameters are reached (position 20on the X-axis in FIG. 4 a) where the background image will contain theoriginal image (possibly histogram equalized but not smoothed). The ACfilter is designed so that for any positive value of D2BR the effectivegain of the details in the final filtered image is at least unity (1×).This is to prevent the filter from smoothing the image in the detailenhancement protion of the AC range (20-100% in FIG. 4 a). Thisrequirement is fulfilled by forcing the Detail Scale Factor (622) to be≧the Background Scale Factor (618) in (see FIG. 6).DDR/(BDR*BG)=D2BR, thus BG=DDR/(BDR*D2BR)  (3)

In accordance with an embodiment of the present invention, by creatingtwo matched tables of values for the variable σ_(R) and D2BRcorresponding to the AC value, the user may tune the AC filter (e.g., ACfiltering algorithm) using one control device, such as a knob (e.g., apotentiometer) or other user-operated selection control. As an example,parameters may be chosen in a specific manner so as to allow a user tocontinuously transition from an image with a low degree of detail (e.g.,filtered detail) to an image with a high degree of detail.

Specifically, in one embodiment, the AC filter may be viewed asoperating as a powerful edge preserving noise filter at one end of thetuning range and as a strong detail enhancing filter at the other end ofthe tuning range. The transition between these two tuning end points maybe achieved in increments (e.g., steps), and the transition may beperceived by the user as continuously moving smoothly from a state ofless AC to a state of more AC (e.g., gradually increasing the finerdetails in the scene of the image). Consequently, the techniquesdisclosed herein may allow for a simpler operation of the infraredimaging system (e.g., infrared camera), for example, with the AC filterset in a manner that is optimal for the particular image during capture.

In one implementation of the AC filtering algorithm, a standard handcontrol device (e.g., such as shown in FIG. 5) having user input controlfeatures serves as a manual control device for manipulating (e.g.,variably controlling) the AC filter. The amount of the AC value may becontrolled by one or more control features, such as a rotatable knob,push buttons, a slide bar, etc. In one embodiment, AC control of theimage (e.g., infrared image) may also allow for some degree of contrastcontrol of the image. In general, rather than a user having to determineand set a number of complex parameters, the AC algorithm may allow asimplified user control (e.g., interface) of these various parameters toobtain the desired image.

As discussed herein, the AC filtering algorithm is adapted to separatethe image (e.g., captured infrared image signal) into at least twoparts: the background image part and the detailed image part. Forexample, a non-linear low pass filter (e.g., a Bilateral filter), withthe variable σ_(R), may be used to separate the image into the twoparts, but this is not limiting. As an example in accordance with anembodiment of the present invention, the captured image may be separatedinto its detailed and background parts (e.g., components) with frequencydecomposition, which may be performed by utilizing, for example, a FastFourier Transform (FFT). In one aspect, frequency decompositioncomprises transforming the image from a spatial domain (e.g., timedomain) to a frequency domain. Details of the image (e.g., detailedimage part) may be defined as components above a specific frequencycomponent (e.g., corresponding to the variable σ_(R) threshold and whichmay be varied similar to a radiometric coefficient). Details of theimage (e.g., detailed image part) may be removed (e.g., separated) fromthe original transform, such as the FFT, which leaves the components forthe background information of the image. The background and detailcomponents may be scaled (e.g., before or after an inverse transform isapplied). Frequency inversion may be performed to transform scaledbackground and detail components from the frequency domain back to thespatial domain (e.g., time domain) by utilizing, for example, an InverseFast Fourier Transform (IFFT). The scaled background and detailcomponents may then be merged (e.g., added) to form an output image.

Furthermore as another example in accordance with another embodiment ofthe present invention, the image (e.g., captured infrared image signal)may be separated into its detailed and background image parts (e.g.,components) with wavelet decomposition. The wavelet decomposition may beperformed in a similar fashion as discussed above for frequencydecomposition, but with the use of wavelet transforms instead offrequency transforms (e.g., wavelet domain rather than frequencydomain).

Additionally as another example in accordance with another embodiment ofthe present invention, spatial low pass filters may be utilized,including linear or non-linear low pass filters. Also, non-linear edgepreserving smoothing methods may be included (e.g., anisotropicdiffusion or variants of median filters). A low pass filter may beapplied to the image signal, for example, to separate the low spatialfrequency component from the image signal, wherein the low spatialfrequency component comprises the background component (e.g., backgroundimage part) of the image. It should be appreciated that a parameter(e.g., a standard deviation for a Gaussian low pass filter) that definesthe amount of smoothing may be determined and varied similar to theradiometric coefficient (the variable σ_(R) threshold). The detailedcomponent (e.g., detailed image part) of the image may be obtained bysubtracting the low pass image component from the original image signalto obtain the details of the image (the detailed component). Thedetailed component and the background component may then be scaled asdiscussed herein and merged (e.g., added) to form an output image.

In accordance with an embodiment of the present invention, FIG. 3 showsa method 300 for processing an image in reference to block 220 of FIG.2. The first part (e.g., background image part) of the captured image isprovided (block 302). As previously discussed, the first part of theimage (e.g., infrared image) comprises the background image part of theimage, which may comprise the low spatial frequency high amplitudeportion of the image resulting from application of the AC filteringalgorithm. The second part (e.g., detailed image part) of the capturedimage is provided (block 304). As previously discussed, the second partof the image (e.g., infrared image) comprises the detailed image part ofthe image, which may comprise the high spatial frequency low amplitudeportion of the image resulting from application of the AC filteringalgorithm. The parameter D2BR (Detail to Background Ratio) defines theratio of the detailed image part to the background image part. Thisparameter may be applied to tune the image in a manner for continuouslytransitioning between an image with a low degree of detail to an imagewith a high degree of detail and/or vice versa.

Therefore, a determination may be made (block 306) as to whether toscale (block 308) the first part (e.g., background image part) of thecaptured image based on the D2BR value (e.g., processing component 110determines whether to scale the first part of the image based on thecorresponding AC value provided by the user control input from controlcomponent 140). If user control input is not sensed (e.g., AC value notprovided) or the first part is not to be scaled, then the first part ofthe image is not scaled and method 300 proceeds to block 310.

A determination may be made (block 310) as to whether to scale (block312) the second part (e.g., detailed image part) of the captured imagebased on the D2BR value (e.g., processing component 110 determineswhether to scale the second part of the image based on the correspondingAC value provided by the user control input from control component 140).If user control input is not sensed (e.g., AC value not provided) or thesecond part is not to be scaled, then the second part of the image isnot scaled and method 300 proceeds to block 314.

The first and second image parts may be merged (block 314) to produce anoutput image for display. As previously discussed, the first and secondparts of the infrared image signal may be separately scaled beforemerging to produce an output image for viewing by a user. This allowsthe imaging system to display an infrared image where fine details arevisible and tunable for improved image quality.

In accordance with an embodiment of the present invention, a table of ACvalues along with corresponding values for the variable σ_(R) and theparameter D2BR may be generated for use by the AC algorithm. Forexample, a user may input a desired AC value (e.g., by increasing ordecreasing the AC value via hand control unit 500 while viewing thedisplayed image), with this provided AC value used to select thecorresponding values from the table for the variable σ_(R) and theparameter D2BR.

As a specific example, processing component 110 may receive the AC valueselected by the user and look up the corresponding values for thevariable σ_(R) and the parameter D2BR (e.g., stored in memory component120) to use to process the image data with the AC filtering algorithm asdiscussed herein. Thus, parameter values for the variable σ_(R) and theparameter D2BR may be generated and stored in a table along with acorresponding range of values for the AC value to tune the image, andthese parameter values may be chosen for example in a manner forcontinuous transitioning between an image with a low degree of detail toan image with a high degree of detail and/or vice versa.

In general, the AC filtering algorithm may be disabled by the user(e.g., switch off or deselect the AC filtering mode) or the user mayselect a desired AC value (e.g., while viewing the displayed image) toprovide a desired image quality for the particular image. For example, alow AC value (e.g., a AC value of 1 on a 1 to 100 scale) may result inthe AC filtering algorithm functioning as an edge preserving spatialnoise filter that produces a smooth image with noise and fine detailsremoved or substantially diminished. As another example, a mid-range ACvalue (e.g., a AC value of 70 on a 1 to 100 scale) may represent atypical setting for many scenes and may result in the AC filteringalgorithm providing a substantial amount of the detail of the image. Asanother example, a high AC value (e.g., a AC value of 100 on a 1 to 100scale) may result in the AC filtering algorithm functioning as anextreme detail booster (e.g., detail enhancer).

The stored AC table values may be adapted to range between two endvalues, such as 1 to 100, with corresponding values for the variableσ_(R) and D2BR (e.g., D_(Range)/B_(Range)). For example, FIGS. 4 a and 4b show embodiments of a graphical representation of how exemplarysettings for the variable σ_(R) and D2BR may vary as the AC inputcontrol (e.g., user input to control component 140 of FIG. 1) ismanipulated to transition from one end point to another, such as scaledvalues 1 to 100 (or 0 to 100%), in a manner as previously discussed.Specifically, FIGS. 4 a and 4 b are plots of two exemplary tables of ACfiltering algorithm values, with AC values along the x axis andcorresponding values of the variable σ_(R) plotted along the left y axis(labeled Definition of Detail and with values of digital units) andcorresponding values of D2BR plotted along the right y axis (labeledDetail to Background Ratio).

The graphs of FIGS. 4 a and 4 b may be viewed as providingrepresentative values (of the variable σ_(R) and D2BR as the AC value isvaried) for two different infrared sensors, respectively. For example,the graph of FIG. 4 b may represent an infrared sensor having more noiseand lower responsivity (e.g., resulting in a lower dynamic range)relative to the infrared sensor represented by the graph of FIG. 4 a.Consequently, the values of the variable σ_(R) and D2BR differ betweenFIGS. 4 a and 4 b as they are related to the dynamic range of theinfrared sensor, with for example the limits of the variable σ_(R) basedon the noise floor (e.g., in digital units) and the dynamic range (e.g.,ten percent or less of the dynamic range) of the particular infraredsensor.

As shown for example in FIG. 4 a, very low AC values may provide noisereduction, such as for low contrast and high noise images, while veryhigh AC values may provide extreme contrast, such as for a low contrasttarget in a high dynamic range scene. In general, for a typical image, auser may find the mid-range AC values to provide the most desirableimage quality and may typically fine tune the AC value within thisrange.

FIG. 6 shows a block diagram 600 for processing an image in accordancewith an embodiment of the present invention. For example, the operationsillustrated in diagram 600 may represent exemplary operations for the ACalgorithm that may be performed by image capturing system 100 (FIG. 1).A digitized signal 602 (e.g., an input frame of an infrared image signalfrom an infrared sensor) is provided and a low pass filtering operation604 (e.g., a non linear Bilateral filtering operation) is performed onsignal 602 and a histogram operation 614 (e.g., to generate a fulldynamic range histogram) is performed on signal 602. Filtering operation604 provides a background portion 605 by using the definition of detailcorresponding to the AC value (e.g., provided by the user). When theBilateral filter is used as the low pass filter AC chosen will definethe details by setting the standard deviation of the parameter σ_(R) inthe Bilateral filter. This effectively defines a maximum amplitude ofsignals that become part of the detail image 621. As an example, theradiometric weights as a function of radiometric distance for thisselected variable σ_(R) may be stored in a lookup table (LUT) forreference (rather than calculating) by filtering operation 604.

For example, referring briefly to FIG. 7, a graph of radiometric weightsas a function of radiometric distance is shown in accordance with anembodiment of the present invention for two exemplary values of thevariable σ_(R). Specifically, the graph illustrates how changing thesettings for the variable σ_(R) (e.g., low pass filter sigmaR parameter)will affect the radiometric weights, with the graph showing plots of twodifferent values for the variable σ_(R). The shape of the curve is thestandard Gaussian as set forth below in equation (3), using the variableσ_(R) (e.g., the standard deviation of the radiometric Gaussian) andwhere D_(R) is the radiometric distance. A higher value for the variableσ_(R) may generally result in greater smoothing and higher amplitudesignals in the detail layer, while a lower value for the variable σ_(R)may generally result in decreasing SNR in the detail image as the valuedecreases to the standard deviation of the noise.

$\begin{matrix}{\mathbb{e}}^{- \frac{D_{R}^{2}}{2\sigma_{R}^{2}}} & (3)\end{matrix}$

A processing operation 606 is performed on background portion 605 fromfiltering operation 604 by application of the histogram provided byhistogram operation 614 to provide histogram equalization and output anequalized background portion 607. The histogram applied, for example,may be delayed by one frame by a delay element 616 and optionallyprocessed 620, such as to ignore small outlying portions (e.g., 0.3% ofthe extreme tails of the histogram which may represent erroneous data).Specifically as an example for processing operation 606, a backgroundLUT may be generated based on the histogram provided by histogramoperation 614 and applied to background portion 605 (e.g., to translatethe dynamic range of the background portion to the desired dynamicrange) to provide the equalized background portion 607.

The background LUT may be modified via an operation 620 for processingoperation 606 of the AC algorithm, such as for example to reduce drasticchanges in the values of the background LUT (e.g., provide low passtemporal filtering, IIR) from one frame to the next. For example,operation 620 may form the background Histogram equalization LUT from acertain percentage (e.g., 90%) of the prior frame's IIR filtered LUT anda certain percentage (e.g., 10%) from the current frame's LUT.

The detail portion 621 (i.e., the second part or, detail image) isgenerated by subtracting (e.g., with a subtractor 610) the backgroundportion 605 provided by filtering operation 604 from signal 602. Thedetail portion is then scaled (e.g., with a multiplier 622) based on theD2BR (Detail to Background Ratio) value, as discussed previously andwith the D2BR value based on the corresponding AC value selected by theuser, to provide a scaled detail portion 623. Equalized backgroundportion 607 is added (e.g., with a summer 624) to scaled detail portion623 to merge the data and provide an output frame 626 of image data.Assuming no scaling of the background portion within the HistogramEqualization step a separate scaling step 618 may be applied (e.g.,using a multiplier) to match the dynamic range of the scaled detailportion. Optionally a final scaling step may be applied to the merged(summed) AC image to scale the signal to the desired output dynamicrange (e.g., the dynamic range required by some video standard).

In general in accordance with an embodiment of the present invention,the AC algorithm (AC filter) may be implemented to split the imagesignal into two parts, with the first part containing the background(low pass) information and the second part containing the detail (finer)information of the image. The second part may be obtained simply bysubtracting (e.g., high pass filter operation) the backgroundinformation from the original image signal. The final output of the ACalgorithm may be generated by scaling the detail information (or thebackground information) so that the detail information will be clearlyvisible in the resulting image regardless of the current dynamic rangeof that image

As a further example, a low pass filter may be used to obtain thebackground information and may be designed such that it will not destroysharp edges in the image. A linear low pass filter will by definitiondestroy edges because it computes a simple weighted mean of neighboringpixels even across sharp edges. Consequently, a bilateral filter may beused in the AC algorithm to dynamically calculate the weights of pixelneighbors based not only on proximity in location (close to the pixelbeing filtered) but also in radiometric proximity. Thus, neighbors withsignificantly higher or lower signal intensity than the pixel beingfiltered may be given zero or close to zero weight.

As another example, the AC algorithm may scale the background and detailinformation linearly to some desired predefined range. For example, atypical setting may divide the available output range (e.g., 256) intotwo approximately equal parts. For moderate dynamic range scenes and forsensors with fairly linear response (e.g., LWIR such as QWIP) thisapproach may be adequate. However, for scenes with large hot objects ordetectors with a more non linear response (e.g., MWIR InSb detectors)this approach may not be adequate. For example, linearly mapping thehigh dynamic range of the background from a 14 to 8-bit domain may makehot objects white and the rest of the background a uniform grey,resulting in an unsatisfactory image quality. Alternatively inaccordance with one or more embodiments, histogram equalization may beapplied to the background information to provide a better distributionof the colors. For example, to preserve some of the radiometricproperties of the image (e.g., it should still be evident that thereexists a very hot object in the scene), a portion (e.g., 25%) of thedesignated background dynamic range may be preserved for a linear orlog-linear mapping.

As another example, a LUT may be created in the histogram equalizationoperation of the AC filter and may include two parts, with one partbeing the standard LUT which scales range proportional to the number ofpixels in that range and another part being a straight linear mapping.For example, typically 25 percent of the range reserved for thebackground will be mapped linearly, with the LUT that is calculatedbeing used for the next frame. Additionally, a low pass filter may beapplied to the LUT to dampen sudden changes in values.

Where applicable, various embodiments of the invention may beimplemented using hardware, software, or various combinations ofhardware and software. Where applicable, various hardware componentsand/or software components set forth herein may be combined intocomposite components comprising software, hardware, and/or both withoutdeparting from the scope and functionality of the present disclosure.Where applicable, various hardware components and/or software componentsset forth herein may be separated into subcomponents having software,hardware, and/or both without departing from the scope and functionalityof the present disclosure. Where applicable, it is contemplated thatsoftware components may be implemented as hardware components andvice-versa.

Software, in accordance with the present disclosure, such as for exampleprogram code and/or data, may be stored on one or more computer readablemediums. It is also contemplated that software identified herein may beimplemented using one or more general purpose or specific purposecomputers and/or computer systems, networked and/or otherwise. Whereapplicable, ordering of various steps described herein may be changed,combined into composite steps, and/or separated into sub-steps toprovide features described herein.

Embodiments described herein illustrate but do not limit the invention.It should also be understood that numerous modifications and variationsare possible in accordance with the principles of the present invention.Accordingly, the scope of the invention is defined only by the followingclaims.

The invention claimed is:
 1. A method of processing an infrared image,the method comprising: capturing the infrared image; processing theinfrared image to provide a background portion of the infrared image anda detail portion of the infrared image; scaling the background portionand/or the detail portion to provide a level of the detail portionrelative to a level of the background portion; receiving a user operatedcontrol signal value from a user controlling two variables with a singlecontrol, wherein the processing and the scaling use filtering and gainvalues, respectively, corresponding to the control signal value; mergingthe background portion and the detail portion after the scaling toprovide a processed infrared image, wherein the merging of thebackground and the detail portion is controlled by the two variables,wherein a first variable of the two variables represents a standarddeviation of the radiometric Gaussian, and wherein a second variable ofthe two variables represents a ratio between a dynamic range of thedetail portion and a dynamic range of the background portion; andstoring or displaying the processed infrared image.
 2. The methodaccording to claim 1, further comprising: performing histogramequalization to the background portion prior to the merging of thebackground portion and the detail portion.
 3. The method according toclaim 2, wherein processing the infrared image comprises: filtering theinfrared image to provide the background portion; and subtracting thebackground portion from the infrared image to provide the detailportion.
 4. The method according to claim 3, wherein the filteringapplies a bilateral filter to the infrared image, and wherein thebilateral filter uses a filter parameter corresponding to the useroperated control signal value.
 5. The method according to claim 4,further comprising: storing a table of values of radiometric weights asa function of radiometric distance for the filter parameter for use bythe filtering.
 6. The method according to claim 1, wherein the scalingis applied only to the detail portion to provide a level of the detailportion relative to a level of the background portion.
 7. The methodaccording to claim 6, wherein the scaling to provide a level of thedetail portion relative to a level of the background portion is based ona gain ratio corresponding to the user operated control value.
 8. Aninfrared image capturing system, comprising: an infrared detectorconfigured to capture an infrared image; a processor; a display; asingle control configured to control both of two variables; and a memorystructure configured to store information to control the processor toperform infrared image processing comprising: processing the infraredimage to provide a background portion of the infrared image and a detailportion of the infrared image; scaling the background portion and/or thedetail portion to provide a level of the detail portion relative to alevel of the background portion; receiving a user operated controlsignal value from a user controlling the two variables with the singlecontrol, wherein the processing and the scaling use filtering and gainvalues, respectively, corresponding to the control signal value; mergingthe background portion and the detail portion after the scaling toprovide a processed infrared image, wherein the merging of thebackground and the detail portion is controlled by the two variables,wherein a first variable of the two variables represents a standarddeviation of the radiometric Gaussian, and wherein a second variable ofthe two variables represents a ratio between a dynamic range of thedetail portion and a dynamic range of the background portion; storing ordisplaying the processed infrared image; and wherein the display isconfigured to display the processed infrared image.
 9. The infraredimage capturing system according to claim 8, further comprising: asensor configured to provide environmental information to the processingcomponent.
 10. The infrared image capturing system according to claim 8,wherein the infrared image processing performed by the processor furthercomprises performing histogram equalization to the background portionprior to the merging of the background portion and the detail portion.11. The infrared image capturing system according to claim 10, whereinthe processing of the infrared image comprises: filtering the infraredimage to provide the background portion; and subtracting the backgroundportion from the infrared image to provide the detail portion.
 12. Theinfrared image capturing system according to claim 11, wherein thefiltering of the infrared image comprises applying a bilateral filter tothe infrared image, and wherein the bilateral filter uses a filterparameter corresponding to the user operated control signal value. 13.The infrared image capturing system according to claim 12, wherein theinfrared image processing performed by the processor further comprises:storing a table of values of radiometric weights as a function ofradiometric distance for the filter parameter for use by the filtering.14. The infrared image capturing system according to claim 8, whereinthe scaling is applied only to the detail portion to provide a level ofthe detail portion relative to a level of the background portion. 15.The infrared image capturing system according to claim 14, wherein thescaling is based on a gain ratio corresponding to the user operatedcontrol signal value.
 16. A computer program product of processing aninfrared image, comprising: a non-transitory computer-readable medium,computer program instructions recorded on the computer readable mediumand executable by a processor for performing a method comprising:capturing the infrared image; processing the infrared image to provide abackground portion of the infrared image and a detail portion of theinfrared image; scaling the background portion and/or the detail portionto provide a level of the detail portion relative to a level of thebackground portion; receiving a user operated control signal value froma user controlling two variables with a single control, wherein theprocessing and the scaling use filtering and gain values, respectively,corresponding to the control signal value; merging the backgroundportion and the detail portion after the scaling to provide a processedinfrared image, wherein the merging of the background and the detailportion is controlled by the two variables, wherein a first variablerepresents a standard deviation of the radiometric Gaussian, and whereina second variable represents the ratio between the dynamic range of thedetailed portion and the dynamic range of the background portion; andstoring or displaying the processed infrared image.
 17. The computerprogram product according to claim 16, wherein the processing of theinfrared image comprises: filtering infrared image to provide thebackground portion; and subtracting the background portion from theinfrared image to provide the detail portion.
 18. The computer programproduct according to claim 16, wherein the method further comprises:performing histogram equalization to the background portion prior to themerging of the background portion and the detail portion.
 19. Thecomputer program product according to claim 18, wherein the filtering ofthe infrared image comprises applying a bilateral filter to the infraredimage, and wherein the bilateral filter uses a filter parametercorresponding to the user operated control signal value.
 20. Thecomputer program product according to claim 19, wherein the methodfurther comprises: storing a table of values of radiometric weights as afunction of radiometric distance for the filter parameter for use by thefiltering.
 21. The computer program product according to claim 16,wherein the scaling is applied, only to the detail portion to provide alevel of the detail portion relative to a level of the backgroundportion.
 22. The computer program product according to claim 21, whereinthe scaling is based on a gain ratio corresponding to the user operatedcontrol signal value.